How to Build a Document Q&A Tool with Python, FastAPI and Azure OpenAI

Build a Lightweight Document Q&A Tool with Python, FastAPI, and Azure OpenAI

Python · FastAPI · Azure OpenAI · about 8 min read

Why build your own document Q&A tool?

You can build a working document Q&A tool with Python, FastAPI and Azure OpenAI in under 150 lines of code, and you don't need a vector database or a heavy framework to do it.

If you've searched for this topic before, you probably noticed the same pattern. Most tutorials jump straight into embeddings, vector stores, orchestration libraries and a dozen config files. That's fine for a production system, but when you just want to understand what's going on, it feels like trying to learn to drive in a Formula 1 car.

So in this post we'll go the other way. We'll build the smallest useful version: upload a file, ask a question, get an answer based on that file. Along the way you'll see how to keep your API keys safe, how to handle file uploads in FastAPI, and how to write one clean function that talks to Azure OpenAI. Once you understand these pieces, adding fancier things later is much easier.

What we're building, in plain words

The idea is simple. A user uploads a PDF or a text file. Our server pulls out the text and cuts it into small pieces, called chunks. When the user asks a question, we pick the few chunks that look most relevant and send only those, along with the question, to the language model. The model answers using just that material.

Why not send the whole document? Two reasons. Models have a limit on how much text they can read at once, and every extra word costs money and time. Sending only the useful bits keeps things fast, cheap and more accurate.

Here's the stack we'll use:

• Python 3.10+ as the language
• FastAPI for the web layer and file uploads
• pypdf to read PDF files
• Azure OpenAI to generate the answers
• python-dotenv to load secrets from a local file

The finished project has just three files: main.py for the API, documents.py for reading and splitting, and llm.py for the model call. Small files, one job each.

Step 1: Set up the project and protect your keys

Start with a clean folder and a virtual environment, then install what we need:

python -m venv venv
source venv/bin/activate   # on Windows: venv\Scripts\activate
pip install fastapi uvicorn python-multipart pypdf openai python-dotenv

The python-multipart package is easy to forget, but FastAPI needs it to accept file uploads. Skip it and you'll get a confusing error the first time you test. If you want to go deeper, the official guide to request files in FastAPI covers every upload option, and Microsoft's Azure OpenAI documentation explains how to create a resource and a model deployment.

Now the part many beginners get wrong: secrets. Never paste your API key straight into your code. The moment that file lands in Git, the key is effectively public. Instead, create a file called .env in the project root:

AZURE_OPENAI_API_KEY=your-key-here
AZURE_OPENAI_ENDPOINT=https://your-resource-name.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-02-01
AZURE_OPENAI_DEPLOYMENT=your-deployment-name

Then add .env to your .gitignore right away, before your first commit. A good habit is to also keep a .env.example file with empty values, so teammates know which variables to fill in. One thing worth knowing: in Azure, the DEPLOYMENT value is the name you gave your model deployment in the portal, not the model's official name. Mixing those up is the most common reason for a "resource not found" error.

Step 2: Read the file and cut it into chunks

Create documents.py. It has three small jobs: pull text out of a file, split it into chunks, and find the chunks that best match a question.

import io
import re
from pypdf import PdfReader


def extract_text(filename: str, data: bytes) -> str:
    """Return plain text from a PDF or a .txt file."""
    if filename.lower().endswith(".pdf"):
        reader = PdfReader(io.BytesIO(data))
        pages = [page.extract_text() or "" for page in reader.pages]
        return "\n".join(pages)
    return data.decode("utf-8", errors="ignore")


def split_into_chunks(text: str, size: int = 800, overlap: int = 150) -> list[str]:
    """Cut text into overlapping pieces so ideas don't get chopped in half."""
    text = re.sub(r"\s+", " ", text).strip()
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks


def top_chunks(question: str, chunks: list[str], limit: int = 3) -> list[str]:
    """Rank chunks by how many question words they share. Simple, but it works."""
    words = set(re.findall(r"\w+", question.lower()))

    def score(chunk: str) -> int:
        return len(words & set(re.findall(r"\w+", chunk.lower())))

    return sorted(chunks, key=score, reverse=True)[:limit]

A few notes on the choices here. The overlap matters more than it looks: if a sentence sits right at the edge of a chunk, the overlap makes sure the next chunk still carries some of it. The ranking function is deliberately basic. It just counts shared words. It won't understand synonyms the way embeddings do, but for a first version it's easy to read, easy to debug, and surprisingly decent on short documents. Also note that scanned PDFs (photos of pages) have no real text inside, so extract_text will return nothing for those. That needs OCR, which is a topic for another day.

Step 3: One clean function to talk to the model

Now llm.py. The goal is to hide every Azure detail behind a single function, so the rest of your app never needs to know which provider is underneath.

import os
from dotenv import load_dotenv
from openai import AzureOpenAI

load_dotenv()

client = AzureOpenAI(
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_version=os.environ["AZURE_OPENAI_API_VERSION"],
)

DEPLOYMENT = os.environ["AZURE_OPENAI_DEPLOYMENT"]

SYSTEM_PROMPT = (
    "You answer questions using only the context provided. "
    "If the answer is not in the context, say you could not find it. "
    "Keep answers short and clear."
)


def ask_model(question: str, context_chunks: list[str]) -> str:
    context = "\n\n---\n\n".join(context_chunks)
    response = client.chat.completions.create(
        model=DEPLOYMENT,
        temperature=0.2,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
    )
    return response.choices[0].message.content

Two details deserve attention. First, using os.environ[...] instead of os.getenv(...) is on purpose. If a variable is missing, the app crashes immediately with a clear message, rather than failing mysteriously later. Second, the system prompt tells the model to stay inside the context. That one sentence does a lot of work in reducing made-up answers, and a low temperature keeps replies steady rather than creative.

Step 4: Wire it together with FastAPI

Finally, main.py. We need two endpoints: one to upload a document, and one to ask a question about it. To keep the demo simple, we'll hold the chunks in memory.

from fastapi import FastAPI, File, HTTPException, UploadFile
from pydantic import BaseModel

from documents import extract_text, split_into_chunks, top_chunks
from llm import ask_model

app = FastAPI(title="Document Q&A")

store: dict[str, list[str]] = {}   # filename -> chunks

MAX_BYTES = 5 * 1024 * 1024        # 5 MB limit


class Question(BaseModel):
    filename: str
    question: str


@app.post("/upload")
async def upload(file: UploadFile = File(...)):
    if not file.filename.lower().endswith((".pdf", ".txt")):
        raise HTTPException(400, "Only PDF and TXT files are supported.")

    data = await file.read()
    if len(data) > MAX_BYTES:
        raise HTTPException(413, "File is too large (5 MB max).")

    text = extract_text(file.filename, data)
    if not text.strip():
        raise HTTPException(422, "No readable text found in this file.")

    store[file.filename] = split_into_chunks(text)
    return {"filename": file.filename, "chunks": len(store[file.filename])}


@app.post("/ask")
def ask(body: Question):
    chunks = store.get(body.filename)
    if not chunks:
        raise HTTPException(404, "Upload this document first.")

    relevant = top_chunks(body.question, chunks)
    return {"answer": ask_model(body.question, relevant)}

Run it with uvicorn main:app --reload, then open http://127.0.0.1:8000/docs. FastAPI generates an interactive page where you can upload a file and try the endpoints with no extra tools.

Notice the guard clauses at the top of /upload. Checking the file type, the size and whether any text was actually extracted takes a few lines, but it saves you from a whole category of confusing bugs and keeps strangers from sending you huge files.

Trying it out, common mistakes, and where to go next

Upload a short PDF through the /docs page, then call /ask with the same filename and a question like "What is the refund policy?" If the answer looks right, you're done. If it says it couldn't find the answer, check whether the question uses different words than the document does. That's the weakness of simple word matching, and it's a good clue about what to improve first.

Here are the problems people run into most often:

• Missing python-multipart. Uploads fail with a vague error.
• Wrong deployment name. Azure wants your deployment name, not the model name.
• Committing .env to Git. Rotate the key immediately if this happens.
• Empty text from scanned PDFs. The file is an image, so you'll need OCR.
• Losing data on restart. Our in-memory store disappears whenever the server reloads.

Once the basic version works, there are natural upgrades. Replace the word-matching function with embeddings, so the search understands meaning rather than exact words. Save chunks in SQLite or a vector store so uploads survive restarts. Return the source chunks alongside each answer so users can verify it. Add streaming so long answers appear gradually, and put simple authentication in front of the API before sharing it with anyone.

The bigger lesson is that a tool like this is just a few ordinary steps: read, split, select, ask. Each step is plain Python. Once you see that, the whole field of LLM apps feels a lot less mysterious, and you can add complexity only where your project actually needs it.

Frequently asked questions

Do I need a vector database to build a document Q&A tool?

No. For a small project you can split the document into chunks, rank them by how well they match the question, and send the best few to the model. A vector database becomes useful when you have many or very large documents, or when you need search that understands meaning rather than exact words.

Why does Azure OpenAI need a deployment name?

In Azure OpenAI you create a deployment of a model inside your resource and give it a name. Your code calls that deployment name, not the model's official name, so a mismatch is the most common cause of a "resource not found" error.

Why does my PDF return no text?

Scanned PDFs are pictures of pages with no real text inside. A library like pypdf can only read actual text, so scanned files need OCR first.

Is it safe to put my API key in the code?

No. Keep keys in a .env file or your host's secret manager, add .env to .gitignore, and rotate the key immediately if it is ever committed.